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ABSTRACT 

The audit report was made in coir.pliance with the 
contractual agreements, legal prescriptions, and official directives 
under the provisions of Title VII of Public Law 89-10, as amended, 
for the establishment and operation of bilingual education programs. 
The audit report (June 12, 1973) is on the Bilingual Education 
Program (in its Uth year of operation) of the Harlandale Independent 
School District. This report gives a critique of the quality of the 
project evaluation, discusses the findings of the project and the 
audit, and confirms or questions the need for program modifications 
proposed as a result of the evaluation. The scope follows the 
suggested U.S. Office of Education audit report areas — (1) 
introductory and general comments concerning the quality and 
significance of the final evaluation report; (2) detailed critique of 
the product and process evaluation conducted for operation and 
management, based on an assessment of the instruments used^ data 
collection procedures, data analysis techniques, and data analysis 
presentation; (3) findings and observations as a result of on-site 
visits and examination of evaluative data with a summary of 
consistencies and discrepancies; (U) recommendations for evaluation 
design revision; and (5) the need for program modification. . (For 
related documents, see RC 007 266, 268.) (FF) 
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EDUCATIONAL ACCOMPLISHMENT AUDIT 
REPORT FOR HARLAtlDALE 



Purpose of the Report 
This Audit Report Is made In compliance with the contractual agreements, legal 
prescriptions, and official directives under the Provisions of Title VII of Public 
Law 89-10, as amended, for the establishment and operation of bilingual education 
programs • This is an Audit Report on the Bilingual Education Program of the Harlandale 
Independent School District* Although this program comprises two public school 
districts, Harlandale and San Marcos, and a university. Southwest Texas State Univer- 
sity, separate evaluations and audits are being performed for the two school districts 
this year in accordance with the changed United States Office of Education directive. 
The program is in its fourth year of opera tionj^ having as its director this year 
Mr. Rene^Gonzalez, The purpose of this report is to give a critique concerning the 
quality of the project evaluation and discuss the comparative findings of the project 
and the audit, and to confirm or question the need for program modifications which 
have been proposed as a result of the project evaluation. 

Scope of the Report 

The Audit Report follows the suggested Audit Report Content Areas received from 

the United States Office of Education which Is a part of the contractual agreement 

between the auditor and the program director as follbws: 

1* Introductory and general comments ccncerning the quality of the 
project evaluation and the comparative findings of the project 
evaluation and the audit, 

2. Detailed critique of the product and process evaluation conducted 

for operation and management, based on an assessment of the instruments 
used, data collectian procedures, data analysis techniques, and 
data analysis presentation. 
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3. Description of the auditor on-site visit findings and their 
correlation with the evaluator^s data and reports; suramary of 
consistencies and discrepancies, and Interpretation of the dis- 
crepancies. 

4. Recommendations* for revisions in the evaluation design, including 
a rationale for each recoinmendation. Since the auditor's ob- 
jectivity can be retained only if the selection of a specific 
corrective action is a local decision, he should provide general 
rather than specific recotnmendations, posing several alternative 
actions or possible sources of assistance to the LEA in correcting 
the deficiency. 

5. Confirmation or questioning of the need for program modifications 
which have been proposed as a result of project evaluation. 

The audit areas described above will be referred to as sections in this report 

with section numbers corresponding to the audit area description numbers. 



SECTION I 

Introductory and General Comments 
Concerning the Quality and Significance 
of the Final Evaluation Report 

The Evaluation Report for 1972-73 has followed the evaluation design* The 
evaluator has accomplished very well the purposes of a bilingual program evaluation, 
which in this instance has been curtailed by federal directives to only the instruc- 
tional component. 

The Evaluation Report for 1972-73 is a remarkable document because of its 
ecientific objectivity, penetrating analyses, logical and lucid interpretations, 
perceptive appraisals, and fundamentally sound recommendations. Project and 
administrative officials would not be remiss to consider the evaluator *s findings 
in their future educational plans. 

This report uses the presentation, analysis, and Interpretation of statistical 
data as its basic format. The extensive use of statistical data is an appropriate 
technique for specific measurement of learning growth and levels of achievement. 
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Many of the expected performance objectives are evaluated by means of realistic 

criterion-referenced quantitative measures. At first glance the statistical data 

may appear to be a bit excessive; however^ these data are most appropriate in an 

adequate evaluative study. 

Concern is often raised by many persons about the relevancy and appropriateness 

of certain evaluation Instruments. This concern has been especially strong in tha 

testing of minority students. The evaluator of this project has used a wide variety 

of measures with several of them administered bilingually or written in Spanish and 

English. The interpretative comments call for appropriate modif icationc taking into 

consideration the target pupil population. The deverslty in individual classrooms 

and in reasonable achievement expectations is evidenced in this report. It is 

such 

obvious that selection and use of tests in a project/as this need constant review 
in search of more valid means of assessment," however^ the utilization of more accurate 
methods of scoring these tests and the availability of longitudinal studies lend 
support to the Instruments now being used. The auditor has noted that the scores 
of the pupils in the program are being compared to their earlier performance under 
the same conditions; therefore^ one can argue that the evaluation results should not 
be critlzed, but that learning opportunities need to be assessed and replanned. The 
evaluator has made several constructive suggestions. 

As verified in an interview with the evalnator> the auditor has ascertained 
that the team of examiners as well as the classroom teachers themselves are the 
most experienced assessment scaff since the proiect began four years ago. This 
experienced team has facilitated a more effective individualized oral and written 
testing program. These strengths along with electronic computer services aid the 
evaluator with more analytical techniques in data interpretation, and certainly are 
commendable additions to the evaluation dimensions. 
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The graphic presentations facilitate comparisons and add greatly to the 
comprehension and interpretation of the data by the concerned reader. Suggestions 
of remediation aspects for the project are more easily obtained from these visual 
aids. The evaluator has made these graphics explicit. 

The auditor notes that Improvements in evaluation techniques usually result 
in magnification to the visibility of instructional results. This clearer view of 
aspects of the project should not result in dissatisfaction with the evaluation but 
on the other hand it should bring into focus certain program aspects in need of re- 
vision and remediation. 

The objectives of the bilingual project are tlie following: (1) to reduce the 
Spanish-surnamed pupils' educational deficit by instructing them in Spanish while 
their command of English is being developed; (2) to enhance their understanding and 
cognitive development in both languages; (3) to give them the advantage of becoming 
literate in both languages; and, (4) to instill in them knowledge of and pride in 
their bicultural heritage. The evaluation report is expected to show some indication 
of how well these objectives have been achieved this year. An on-going assessment 
of the project's objectives as to their practicality needs to be part of the program 
itself. 

Objectivity and comprehensiveness appear to be major characteristics of this 
report. The recommendations are educationally sound, clearly stated, and are based 
on much empirical evidence. 

A project description, beginning on Page 1 of the evaluation report^ is an 
accurate account. There are four objectives of the bilingual program which have 
appropriately been placed as the first part of this section. These intended out- 
comes are the criterion on which the entire project should be evaluated. Several 
comments through this audit speak to the adequacy of the evaluation in obtaining 
these goals. 



Tabic I, "Teachers, Schools, and Summary of Pupil Data," shows teacher 
assignments by schools, grade levels, and class sizes. Of the eight schools with 
55 teachers in the program, two schools involving lA teachers employ team-teaching 
arrangements- There are 1517 children In grades K-5 in eight of Harlandale's 
fifteen elementary schools Involved in the project. Ninety-nine percent of these 
children have Spanish-surnames with a majority from homes in which Spanish is the 
dominant language. 

In spite of the fact that the Consortium, which operates the bilingual project, 
encompasses two cities some fifty miles apart, all aspects of the program have 
functioned well. Since each of the two public school districts Involved has had 
its own coordinator of the learning component, very little, if any, detrimental 
effect on the instructional program has resulted because of one district being 
fiscal agent, while offices for the project director and the evaluator along with 
the university facilities and the other school district were in a separate city. 

As has been mentioned in the Interim Report, a major change in the program this 
year is the assumption of financing for the second grade as well as the first by the 
local district. This is even greater evidence of the commitment by the school 
district to bilingual education. 

A major management personnel change has been made this year — a new project 
director, Mr. Rene Gonzalez, has replaced Mr. Carlos Rodriguez. The auditor concurs 
with the evaluator that increased efficiency as well as community involvement by a 
new full-time director has enhanced the total bilingual project. 

Under the direction of last year's project director, Mr. Carlos Rodriguez, 
Southwest Texas State University's bilingual-bicultural teacher-training center has 
greatly contributed to this bilingual project. It is easily observed that in 
addition to the pre-service training of teachers, this program has contributed to 
the efficiency of the testing and instructional aspects of the project. 



SECTION II 

Detailed Critique of the Product and Process 
Evaluation Conducted for Operation and Management, 
Based on an Assessment of the Instruments Used, 
Dnta Collection Procedures, Data Analysis 
Techniques, and Data Analysis 
Presentation 

It has been observed that this year's federal guidelines curtailed evaluation 
to the Instructional conponent, thcrefore^other components, such as staff develop- 
ment (p. 1), were dealt with only incidentally as they affect instruction. Also, 
directives accorded that standardized tests form the baci;bone of the evaluation this 
year and other performance objectives have been eliminated from the deslRji. It Is 
the auditor's belief that a total evaluation of achievement should Include various 
subjective assessments as well,' however^ the Instruments selected for this year's 
project evaluation are vexry feasible under the federal directives • 

The auditor's critique and comments are limited to his knowledge and experiences 
with the project since the middle of the school year (February 7, 1973). The Illness 
of the former auditor. Dr. Guy C. Pryor, necessitated a change. As time permitted, 
every school campus, school principal, and project administrative personnel were 
visited. These visits, test samples, test Instruments, as well as classroom processes 
are bases for this report. More Information will be related later In this report 
under the section on ''On-Slte Visits". 

The project evaluation was conducted very effectively and efficiently on schedule 
as planned and supervised by the evaluator. The klndergdrten and first grade students 
were evaluated Individually on vocabulary by a trained testing team composed of 
Southwest Texas State University's intern student teachers. The regular classroom 
teachers administered readiness and achievement tests In cooperation with the 
evaluator. The auditor has observed that the Individual examiners are quite 
experienced, due perhaps to the fact that many have worked In the program for several 
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years. The project evaluator can be commended for a very highly organized and 
well-planned operation and management of testing procedures. This function was 
carried out in a most business-like manner. Further comments will follow as the 
auditor critiques the evaluation product and process. 

Scores for Peabody tests and those for all other evaluation instruments 
administered are put on cards and electronically processed • The process of returning 
feedback on these scores to project teachers not only allows for early diagnosing 
of pupil v;eaknesses but also facilitates earlier corrective action. The auditor 
observes this as a significant factor In the learning process — especially as the 
number of pupils increases, more data is gathered and disseminated in the evaluation 
process. 

Table II, "Peabody Picture Vocabulary Tests, compares 1971-72 dominant and 
balanced language pupil mean scores with those of 1972-73 for kindergarten and first 
grade pupils. The significant evidence of handicaps In concept formulation in both 
sets of figures presented magnifies the need for teachers to implement extensive 
measures to reduce this deficit. 

A very real accomplishment for the bilingual kindergarten was observed when 

fall Peabody mean scores for first graders were compared. Forty-three bilingual 

kindergarten pupils who attended Collier school last year were ahead of two groups 

of first graders who had been in non-bilingual kindergarten classrooms or had not 

been in kindergarten at all (Table III) , This is commendable and represents the 

fulfillment of project objectives to reduce concept deficits. 

Peabody 

When fall and spring scores of pupils in kindergarten and first grade were 
coraparec^ greater numbers achieved expectable gains in English than in Spanish (Table 
IV). Specific classrooms had differences of slb much as 58% variation. Although 
gratifying gains were shown in mean increase on Peabody scores, variation in mean 
scores between classrooms showed a rather wide range of success (Table V). It is 



stressed by the auditor that the coordinator follow closely the recommendations by 
the evaluator to study Table V and take appropriate action with improvement in 
teaching of concept development. It is satisfying to observe that overall analysis 
of Peabody scores, including quartile assessment (Table VI), shows that teachers in 
the project are successfully working to remedy the concept deficit. 

The Metropolitan Readiness Test and Metropolitan Achievement Test results, 
(Tables VII-VIII), indicate a high percentage of success toward behavioral objectives 
for kindergarten and first grade students. The auditor concurs that the bilingual 
program Is successfully fulfilling the objective of reducing educational deficit. 
The graphic presentations give a clear picture of explanations and findings — es- 
pecially the graphs illustrating variability and growth at three quartile levels 
facilitate comparisons, (Tables XIII-XIV), These are commendable procedures* 

The evaluator reports (p. 27) that in grades 2-5 generally the amount of gain 
is good for Metropolitan Achievement Test areas of word discrimination, language, 
and math computation, (Tables IX-XII) . A few percentages in these areas, as well 
as social studies and science^ had lesser degrees of success. 

When quartile scores were analyzed, (Tables XV-XVIII), a large variability 
existed between classrooms as to the percentage of pupil success* Also, a large 
variability existed between classrooms as to the area(s) of pupil success. Medians 
in word discrimination, math computation, and language are usually shown higher 
than those in word knowledge, reading comprehension, and math problem solving/concepts. 
The evaluator has high-lighted the evidence produced by the data that there is a 
great need for extensive consultation among the coordinator and teachers. Determining 
reasons for expertise in some areas and a sharing of this expertise with fellow 
teachers, as well as further college work or in-service training for needy teachers^ 
are suggested. The auditor concurs that the coordinator approach this recommenda- 
tion as soon as possible In a most professional manner. 
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It has been noted that the pupils In the second grade are even with national 
norms but fall further behind the national norms on these achievement tests each 
year that they are in school. The evaluator points out (p. 28) that there is no 
achievement test on the market at this time which is constructed for or normed for 
the pupil population represented in this project. It is recommended by the auditor 
that, whether bilingual projects are continued on a locally or federally funded basis, 
some type of standardized achievement assessment device bo sought. The school dis- 
trict should keep a constant vigil for such an instrument. 

According to the evaluator, the Inter-Americana Spanish reading teat, the 

Prueba de Lectura, was administered to grades two, three, four, and five, (Table XIX). 

Only raw scores were used to compare progress. The test was reported as being 

difficult (pp. 28-29), but was the only one available at the coinmencement of this 

project and is presently used so that a longitudinal study is possible next year. 

The evaluator is to be commended for the clear and accurate summative analyses and 

comments. Although results are generally good, there is a wide variation reported in 

some mean scores, (Tables XX-XXI). The coordinator should study the recommendations 

of the evaluator so that the level of achievement expected is gained in all class- 

rooms. Perhaps a less difficult reading test might be employed in the project after 

the longitudinal study is made next year. 
Results of 

/evaluation in grades 1-4 with the Bilingual Education Program locally -constructed 
tests in social studies and science were reported as rather good. (See Table XXII.) 
Half of the classrooms were given the English version of the test, and the other half 
were given the Spanish version. Means and standard deviations were derived for fall 
and spring scores to determine whether accomplishment was higher in English or Spanish, 
(Tables XXIII-XXIV) . The statistical ^-test of variance between English and Spanish 
performance planned for next year should prove interesting. The auditor recommends 
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that the Bilingual Education Program test forms be clearly printed so that children 
will be ablG to distinguish the pictures withhut difficulty. 

One factor directly related to achievement by the pupils was t:he amount of time 
teachers spent teaching various areas of learning. The coordinator, curriculum 
writer/ and evaluator have taken much time to report this data in Tables XXV, XXVI, 
and XXVII. The successful or poor test results along with the improvement levels 
over last year correspond very closely with the percentage of time spent on a 
particular subject area. The percentage of time the language was used to convey the 
material correlated highly with the degree of success in performauce as well. The 
coordinator would do well to study these tables and follow the evaluator 's suggestions. 

The data reported in Tables XXVIII and XXIX, "inferred Self-Concept Scales/' 
presents creditable evidence that the bilingual education program is indeed helping 
many Spanish-surnamed pupils to achieve a more positive self-image. The auditor 
commends the teachers a. id scorers on the amount of time and effort obviously expended 
in evaluating this phase of the project. Due to the fact that this type of evaluation 
is highly subjective, the auditor recommends that a general meeting of all teachers 
evaluating with these forms have a brief in-service workshop with Dr. Elizabeth 
McDaniel, author of the instrument, at the beginning of next school year. Perhaps 
a better understanding of how to score the self-concept scale would reassure validity 
and reliability of this evaluative device. 

The bilingual project is to be commended in general for its attempts to prevent 
drop-out of students. Certainly, success in this endeavor will affect the program 
positively. The two-day conference planned by Mr, Carlos Rodriguez in November 
which 45 central and south Texas administrators attended is a positive approach. 

The evaluator has done commendable work in comparing control groups taught in 
traditional classrooms to those of experimental bilingual groups this year for the 
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first time. The comparisons were made in second, thirds and fourth grade classes. 
Two severe limitations existed in the study, and realistic assessment reported 
(pp. 53-54) that there seems to be no advantage indicated in either group, although 
five of the seven statistically significant comparisons favored the experimental 
group. Significant variables, especially the teacher variable, could not be controlled 
which contributed to doubtful validity of the study. As it appears feasible, more 
studies of this kind are recommended. 

Other components vitally associated with the learning process and evaluative 
process and alluded to only briefly in this report appear to have had a positive 
direction this year. The auditor certainly concurs that the call for devoting of 

more time to concept development and reading comprehension this year was warranted 

and well-founded. The coordinator and supervisors have been working diligently to- 
ward a stronger teaching staff. (See p. 56.) Parental contacts with the schools 
and their interest is increasing. All of these facts are creditable signs for a 
successful program. 

SECTION III 

Findings and Observations as a Result of On-Site Visits 
and Examination of Evaluative Ejata with a Summary 
of Consistencies and Discrepancies 

The activities engaged in by the auditor on on-site visits included: 

1. Employed by Mr. Rene Gonzalez, project director, and Dr. Helene 
Harrison, project evaluator in early February at which time duties 
of the auditor were discussed. 

2. Attended several follow-up conferences with evaluator and project 
director to discuss relationships between evaluating and auditing^ 
processes. 

3. Examined and approved all tests, rating forms and scales, and 
evaluation instruments. 

4. Interviewed teachers and principals. 
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5. Observed post-tGSt administrations. 

6. Conferred with testing-team members as well as teachers. 

7. Sampled most adequately the test results, opinicnnaires, rating 
scales, data, and data processing techniques. 

8. Conferred with the project coordinator, curriculum writer, and 
other resource personnel. 

9. Examined samples of curriculum and instructional materials used 
in the project. 

10- Attended meeting LEA administrators , project personnel, 

elementary principals, and teachers concerning the implementation 
of bilingual projects. 

11. Conferred with LEA administrators concerning plans. 

12* Visited several bilingual classrooms. 

13. Went over the evaluation and audit report with evaluator and 
other personnel. 

The thoroughness of the discussions between the evaluator, the auditor, the 
director, coordinator, and other personnel has reflected a minimum of discrepancies 
between the evaluation design and findings and the auditor's checking and concluciors. 
The consistencies are very numerous indeed. As mentioned in Sections I and IIj 
numerous specifics have been identified which need not be repeated. The many appraisal 
statements of the evaluation and its findings in these sections reflect aspects of 
the consistencies and discrepancies. The completeness of the evaluative data with 
its careful analysis and objective interpretation have reduced the discrepancies to 
a minimum. Major concerns are raised relating to the recommendations of the evaluator 
on Page 58 of the report. These recommendations are well-founded in the evaluation 
data. 

The first recommendation concerning minimizing of pupil drop-'out-of-the-prograra 
is well founded. Certainly all administrators need be cognizant of maintaining more 
pupil continuity. 
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The suggestions for the continuing need for recruitment of bilingual teachers 
are well established • 

The recommendations for more effective time allocation in team-teaching should 
be studied and Implemented. Concept development in both languages is apparently in 
need of further study. 

The suggestion — "the Importance of spending less time on phonics, spelling, and 
sound discrimination and more time on reading comprehension in both English and 
Spanish language arts.^' — is strongly indicated by evidence. A limited use of 
cuadernos as a suggestion should be heeded. 

Recommendations by the evaluator concurred to by the auditor stated that the 
coordinators should scrutinize the tables in the evaluation report to determine in- 
stances of teachers' strengths and weaknesses. A follow-up with planned improvement 
procedures and/or sharing of expertise by individual teachers is suggested. 

Once again, the auditor agrees unequivocally with these educationally sound 
recommendations. They are clearly stated, completely valid, and should be implemented 
in order to make the project the most valid possible. 

SECTION IV 

Recommendations for Revision of 
the Evaluation Design 

Only one minor recommendation for revision within the evaluation design is 
suggested by the auditor. If a standardized test with regional and/or natlcniil 
norms is available, it should replace tests now being used with only raw scores. 
Perhaps this would be more feasible after the project has operated for five years 
and appropriate longitudinal studies attempted. 

With the scope of the evaluation for this project being limited by federal 
directives this year, the auditor recommends no major changes at this time. Another 
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look at the uniformity of scoring by various teachers on the pupil self-concept 
scale might be considered. 



SECTION V 

Confirmation or Questioning of the Need for Program 
Modifications which Have Been Proposed as 
A Result of Project Evaluation 

The auditor of this project has found no cause to question the recotirmcndatlons 
for program revisions. A strong endorsement of these suggestions Is appropriate. 

The classroom teacher is the most Important factor effecting the outcome of 
the bilingual J blcultural educational project. The auditor most sincerely confirms 
that any recommendations for performance Improvements of the classroom teacher 
should be re-emphasized. If the behavioral objectives of the program are to be 
achieved in a more effective manner, the teacher must constantly plan proper 
activities which will give children opportunities to practice the intended terminal 
behaviors. 

It is most proper that the suggested modifications based on revelations of the 
project evaluation be implemented with all haste. 

Summary of Recommendations 
and Comments 

1. The Final Evaluation Report for 1972-73 has been limited in scope 
as compared to former years, yet is very comprehensive and most 
specific. 

2. The evaluation is accurate, and the data have been propoerly 
analyzed and Interpreted. 

3. The recommendations and suggestions made by the evaluator are 
derived from thorough assessments and are educationally sound 
and administratively feasible. They should be studied by all 
educators associated with the project and the n implemented. 

A. The LEA administrators should be cognizant of a continuing need 
for pupil retention in the project, as well as of a constant need 
for bilingual teacher recruitment. 
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5. Although there is a continued effort for improving the learning of 
concepts in both English and Spanish, a more concerted effort 
should be made. Also, Spanish-language teaching should be given 
more time, especially on lower grade levels. 

6. According to suggestions, more time should be spent on reading 
comprehension in both English and Spanish and less time on phonicSj, 
spelling, and sound discrimination. 

7. As has been stated earlier, a careful look at tables in the project 
evaluation should be made to determine weak and strong areas for 
respective teachers. Remedial or refresher measures and/or a 
sharing of expertise should be followed for and by these teachers. 

8. The a^.inlnlstrators of Harlandale Independent School District and 
Soulhwest Texas State University along with the San Marcos Inde- 
pendent School District are to be commended for having initiated 
and madci provisions for a bilingual instructional program through 
these four years with improvements each year, 

9. The auditor has observed that the evaluator has been very res- 
ponsive to suggest lonSs and discussed these in a professional 
manner. Each recommendation has been responded to promptly x^ith 
appropriate and satisfactory dispatch. The Evaluation Report is 
highly commended. 

10. The project has mucT\ Jnerlt and should be funded in somu manner 
after the projected five-year period. 
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